Method of forming an image based on a plurality of image frames, image processing system and digital camera

ABSTRACT

Image fusion based on a modified method of frame averaging for noise removal by partly averaging over images having a smaller resolution than the desired resolution of the de-noised image. The set of images which are summed for averaging out noise consists of two subsets. The first set of images has a resolution (in terms of number of pixels) being smaller than the resolution of the images in the second set. The resolution of the images in the second set is the resolution of the “high-definition” de-noised output image. The lower resolution images are up-sampled by scaling their pixel numbers to that desired output image. The gradation of the first set images is also adapted to avoid intensity saturation (flare) due to summation. Image fusing is also done in fourier space using the high frequency components from the higher resolution images and the lower ones from the lower resolution images.

The invention relates to a method of forming a combined image based on aplurality of image frames.

The invention also relates to a system for processing arrays ofintensity values, each array being suitable for representing an imageframe at a resolution corresponding to the number of intensity values inthe array.

The invention also relates to an imaging apparatus, e.g. a digitalcamera.

The invention also relates to a computer program.

International patent application PCT/EP2005/052121 was filed before andpublished under number WO____/______ after the date of filing of thepresent application, and is thus comprised in the state of the artaccording to Art. 54(3) EPC only. It describes a method of forming acombined final image from a plurality of image frames, including thesteps of obtaining a first and at least one further array of pixelvalues, each array of intensity values encoding light intensity levels aeach of a respective number of pixel positions in the respective imageframe, the number determining the spatial resolution of the image frameconcerned. A set of derived arrays of intensity values is generated,each derived array being based on a respective one of the obtainedarrays of intensity levels and encoding light intensity levels at eachof a common number of pixel positions in at least a region of overlap ofthe respective image frames. An array of combined intensity values isgenerated. Each element in that array is based on a sum of intensityvalues represented by a corresponding element in each of the respectivederived arrays of intensity values. An array of intensity valuesencoding the combined final image is provided, the array being based onthe array of combined intensity values. A first array of intensityvalues encoding at least the region of overlap at a higher resolutionthan the further arrays of intensity values is obtained. An array ofintensity values encoding at least the region of overlap in the combinedfinal image at a higher spatial resolution than the further arrays ofintensity values is provided. The array of intensity values encoding thecombined final image is based on a sufficient number of intensity valuesin the first array of intensity values to encode the region of overlapat a higher resolution than the further arrays of intensity values.

Forming a combined image by adding a plurality of image frames at leastpartially depicting the same region has the effect that the region ofoverlap has a higher Signal-to-Noise Ratio (SNR) in the combined imagethan in the individual image frames. However, in an image processingsystem, intensity values assume one of a range of discrete values, thenumber of which is determined by the number of bits by which the valuesare represented. This in turn is determined by the dynamic range allowedby the format in which the combined image is displayed, e.g. the JPEGstandard or the resolution of a computer display. If the sum of theintensity values corresponding to a pixel in the respective image framesexceeds the maximum allowed by the range of discrete values, the sumvalue is clipped to stay within the range. If this happens for manyintensity values in the array of intensity values representing thecombined image, the combined image appears over-exposed.

It is an object of the invention to provide a method, system, imagingapparatus and computer program of the types indicated above, forproviding in an efficient manner a combined image that has a relativelygood SNR and little or no over-exposure.

This object is achieved according to the invention by providing a methodof forming a combined image based on a plurality of image frames,including:

obtaining a first set of at least one array of intensity values forrepresenting an image frame at a resolution corresponding to the numberof intensity values in the array, and

obtaining a second set of at least one array of intensity values forrepresenting an image frame at a resolution corresponding to the numberof intensity values in the array,

wherein the combined image is represented by a final array of intensityvalues,

wherein at least some of the intensity values in the final array areeach obtained by executing a step of summing an intensity value fromeach of at least one array of intensity values based on at least onearray of intensity values in only the first set and an intensity valuefrom each of at least one array of intensity values based on at leastone array of intensity values in only the second set, wherein, prior toexecuting the summing step, only the intensity values of the arrays inthe first set are mapped from a scale within a first range to a scalewithin a second range.

Because at least some of the intensity values in the final array areeach obtained by executing a step of summing an intensity value fromeach of at least one array of intensity values based on at least twoarrays of intensity values, the SNR is improved. Because the intensityvalues of the arrays in the first set are mapped from a scale within afirst range to a scale within a second range prior to executing thesumming step, it is possible to use the full dynamic range allowed bythe representation of the intensity values without going beyond the endof the scale on which they are represented. For this purpose, the secondrange is different from the first range. Because only the intensityvalues of the arrays in the first set are mapped, the method isrelatively efficient.

An embodiment of the invention includes obtaining a first set and asecond set arranged such that the image frames represented by the arraysin the first set are represented at lower resolutions than the imageframes represented by the arrays in the second set

This has the effect of increased efficiency, as relatively few intensityvalues are mapped from the scale within the first range to the scalewithin the second range.

In an embodiment, the at least one array of intensity values based on atleast one array of intensity values in only the first set containscoefficients in the spatial frequency domain, the at least one array ofintensity values based on at least one array of intensity values in onlythe second set contains coefficients in the spatial frequency domain andthe intensity values in the final array are formed by coefficients inthe spatial frequency domain,

wherein at least one lower-order coefficient in the final array isobtained by summing an intensity value from each of the at least onearray of intensity values based on at least one array of intensityvalues in only the first set and an intensity value from each of atleast one array of intensity values based on at least one array ofintensity values in only the second set,wherein at least one higher-order coefficient in the final array isobtained on the basis of only arrays of intensity values based on thesecond set.

This is a relatively efficient way of obtaining a combined imagerepresented at a relatively high resolution on the basis of a first setof arrays representing image frames at a lower resolution and a secondset of arrays representing image frames at a higher resolution.Interpolation or similar techniques to increase the resolution of theimage frames represented by the arrays of the first set is not required.Instead, the information in the higher resolution image framesrepresented by the second set is used to generate a relativelyhigh-resolution combined image, whereas summation of the lower-ordercoefficients serves to decrease the perceptible noise in the image.

In an embodiment, at least some of the arrays of intensity values in thefirst and second set are obtained by reading out measurement values froman image-capturing device comprising an array of light-sensitive cells,wherein each intensity value in the final array is based on at least oneintensity value in an array comprised in the second set.

Because the arrays in the first set represent image frames at a lowerresolution they contain fewer intensity values. Thus, the time to readout the measurement values is reduced. This allows the image framesrepresented by the first and second sets of arrays to be read out inquick succession, decreasing the effect of camera shake or movement inthe scene that is captured. Because each intensity value in the finalarray is based on at least one intensity value in an array comprised inthe second set, the effect of decreased blur due to movement is notobtained at the expense of the resolution of the combined image.

An embodiment includes determining an upper limit of the second range atleast partly in dependence on the number of arrays of intensity valuesin the second set.

Thus, the risk of an over-exposed combined image is reduced.

In an embodiment, at least one of the arrays of intensity values in thefirst set is obtained by obtaining a plurality of arrays of intensityvalues for representing an image frame at a resolution corresponding tothe number of intensity values in the array, and by summing an intensityvalue from each of the plurality of arrays to obtain a correspondingintensity value in the at least one array in the first set.

Thus, an array representing an image that is the sum of a plurality ofimage frames is scaled. This has the effect of decreasing the amount ofscaling that has to be done, making the method more efficient. Inaddition, random noise over the plurality of arrays that are summed toform an array in the first set is filtered out by means of the addition.

In an embodiment, at least one of the arrays of intensity values in thefirst set is obtained by obtaining a plurality of arrays of intensityvalues for representing an image frame at a resolution corresponding tothe number of intensity values in the array, wherein the method furtherincludes

summing an intensity value from each of the obtained plurality of arraysto obtain a corresponding intensity value in an intermediate combinedarray, anddetermining an upper limit of the second range at least partly independence on at least one intensity value in the intermediate combinedarray.

Thus, the appropriate extent of the second range can be determinedrelatively accurately, since it is based on an array of intensity valuesthat is quite representative of the final array. This embodiment is alsorelatively efficient, since it does not require an analysis of each of aplurality of arrays in the first set.

In an embodiment, at least the arrays of intensity values in the firstset are obtained by obtaining a plurality of arrays of intensity valuesfor representing colour image frames in a first colour space, andapplying a transformation to a plurality of arrays of values in a secondcolour space, wherein, in the first colour space, an image frame isrepresented by parameter value combinations, each parameter indicatingthe intensity of one of a plurality of colour components, whereas, inthe second colour space, an image frame is represented by parametervalue combinations, one parameter of the combination indicating a hueand at least one of the other parameters being indicative of lightintensity.

This embodiment has the advantage that the mapping from the first scaleto the second scale need be carried out on fewer arrays of intensityvalues. Instead of separate arrays of intensity values for each colourcomponent, or arrays of intensity value combinations, only the array orarrays of parameter values indicative of light intensity in the secondcolour space, or arrays derived based thereon, need be processed. Thecolour information is contained in an array of parameter valuesindicating hues, which need not be scaled to prevent saturation of thecombined image.

According to another aspect, there is provided in accordance with theinvention a system for processing arrays of intensity values, each arraybeing suitable for representing an image frame at a resolutioncorresponding to the number of intensity values in the array,

wherein the system is configured to retrieve a first set of at least onearray of intensity values and a second set of at least one array ofintensity values, the arrays in the first set and arrays in the secondset representing respective image frames, and to form a final array ofintensity values representing a combined image,

wherein the system is configured to obtain each of at least some of theintensity values in the final array by executing a step of summing anintensity value from each of at least one array of intensity valuesbased on at least one array of intensity values in only the first setand an intensity value from each of at least one array of intensityvalues based on at least one array of intensity values in only thesecond set, and

wherein the system is configured to map, prior to executing the summingstep, only the intensity values of the arrays in the first set from ascale within a first range to a scale within a second range.

According to another aspect, there is provided in accordance with theinvention an imaging apparatus, e.g. a digital camera, comprising aprocessor and at least one storage device for storing a plurality ofarrays of intensity values, wherein the imaging apparatus is configuredto execute a method according to the invention.

The imaging apparatus makes relatively efficient use of digital signalprocessing capacity. In particular, because not all arrays of pixelvalues are scaled, the amount of values to be retrieved from a look-uptable implementing the mapping function is relatively low.

According to another aspect of the invention, there is provided acomputer program, including a set of instructions capable, whenincorporated in a machine-readable medium, of causing a system havinginformation processing capabilities to perform a method according theinvention.

The computer program can be run on a general-purpose computer forpost-processing of captured images, or it can be provided in the form offirmware for an image-capturing device such as a digital camera.

The invention will be explained in further detail with reference to theaccompanying drawings, in which

FIG. 1 illustrates schematically a digital camera equipped to implementa method of forming a combined image;

FIG. 2 illustrates schematically a first embodiment of a method offorming a combined image;

FIG. 3 illustrates schematically a second embodiment of a method offorming a combined image;

FIG. 4 illustrates schematically a third embodiment of a method offorming a combined image; and

FIG. 5 illustrates schematically a fourth embodiment of a method offorming a combined image.

FIG. 1 illustrates some components of a digital camera 1 as an exampleof an imaging apparatus adapted for implementing the methods describedbelow. Other examples of suitable imaging apparatus include scanners andphotocopying apparatus. Because the methods of forming a combined imagerequire relatively little processing capacity, it is advantageous toapply them in the digital camera 1.

The digital camera 1 includes a lens system 2 for focussing on one ormore objects in a scene that is to be represented by a combined image.When a shutter 3 is opened, the scene is projected through an aperturein a diaphragm 4 onto a photosensitive area of an image-capturing device5. Instead of the shutter 3, an electronic shutter implemented bysuitable control of the image-capturing device 5 could be used. Theshutter time is controllable, as is the diameter of the aperture. Theimage-capturing device 5 can be a device implemented in ComplementaryMetal-Oxide Semiconductor (CMOS) technology, or a Charge-Coupled Device(CCD) sensor, for example. The photosensitive area of theimage-capturing device 5 is divided into areas occupied by pixel cells.Each pixel cell includes a device for generating a signal indicative ofthe intensity of light to which the area that the pixel cell occupies isexposed. An integral of the signal generated by a device is formedduring exposure, for example by accumulation of photocurrent in acapacitor. Subsequent to exposure for the duration of an exposure timeinterval, the values of the integrals of the generated signals are readout row by row.

The (analogue) values that are read out are provided to anAnalogue-to-Digital (A/D-)converter 6. The A/D converter samples andquantises the signals received from the image-capturing device 5. Thisinvolves recording the intensity values on a scale with discrete levels,the number of which is determined by the number of bits of resolution ofthe digital words provided as output by the A/D converter 6. Thus, theA/D-converter 6 provides as output an array of intensity values recordedon a scale occupying a first range. Each intensity value is associatedwith a particular pixel position in an image frame, corresponding to aphotosensitive cell or a plurality of adjacent photosensitive cells. Inthe latter case, the values read out from the image-capturing device 5are preferably obtained by “binning” the values corresponding to aplurality of adjacent photosensitive cells. The areas to which the“binned” values correspond may overlap.

Each exposure of the image-capturing device 5 thus results in an arrayof intensity values representing an image frame. As will be explained inmore detail below, the intensity values of one or more arrays may bemapped to a different scale occupying a second range by a Digital SignalProcessor (DSP) 7. In certain embodiments, the DSP 7 is also suitablefor performing such operations as interpolation between pixel values andoptionally compression of the image. It may also carry out atransformation of the intensity values to the spatial frequency domain,such as a Direct Cosine Transform (DCT).

Arrays of intensity values are stored in a storage device 8. The storagedevice can be any usual type of storage device, e.g. built-in flashmemory, replaceable flash memory modules, an optical disk drive or amagnetic disk drive.

Capturing and processing of images is carried out under control of amicroprocessor 9, which issues commands over a bus 10. Themicroprocessor 9 is assisted by a co-processor 11 in the illustratedembodiment. The co-processor 11 is preferably a digital signal processorfor performing image compression, for example in accordance with theJPEG standard. The microprocessor 9 comprises a volatile memory and hasaccess to instructions stored in Read-Only Memory (ROM) module 12. Theinstructions provide the digital camera 1 with the capability to performa method of forming a combined image by adding a plurality of capturedimage frames, which method is carried out under the control of themicroprocessor 9.

Other components connected to the bus 10 include an input interfacemodule 13 for receiving user commands, and an output interface module 14for returning status information. In the illustrated embodiment, amotion sensor 15 is present for sensing and measuring movement of thedigital camera 1. In other embodiments, a series of image framescaptured in rapid succession is analysed to determine the amount and/ordirection of movement of the digital camera 1. In addition, the digitalcamera 1 comprises an exposure metering device 16 and a flash driver 17for directing the operation of a flash (not shown).

In use, a user issues a command to form a single image of a scene, whichis passed on to the microprocessor 9 through the input interface module13 and the bus 10. In response, the microprocessor 9 controls thedigital camera 1 such that a plurality of underexposed image frames orimage frames with a high ISO setting are captured. A high ISO settingmeans that the sensitivity of the image-capturing device 5, calibratedalong the linear film speed scale according to international standardISO 5800:1987 is set to a high level. The captured images representrespective scenes that overlap at least partially. Each image frame,specifically each colour component of an image frame, is represented byan array of pixel values. Each pixel value corresponds to the lightintensity of the associated colour component over an area associatedwith a pixel. Given that each area associated with a pixel correspondsto a part of the area of the image-capturing device 5, which isconstant, the number of intensity values contained in an arraycorresponds to the spatial resolution of the image frame. This is alsothe case where the intensity values are coefficients in the spatialfrequency domain, since the inclusion of more values in an arraycorresponds to the presence of coefficients of a higher order.

To obtain the sequence of individually underexposed image frames, themicroprocessor 9 determines a desired exposure for a final image to beformed on the basis of the image frames. This exposure is divided overthe image frames. The desired exposure can be determined from user inputor automatically on the basis of one or more values obtained from theexposure metering device 16. Exposure levels for each of the imageframes result in settings of the diaphragm 4, shutter speed and flashintensity. In addition, the microprocessor 9 determines amplificationlevels for the signals read out from the image-capturing device. Thesedetermine the range of values within which the intensity values in thearrays representing the image frames lie. The number of bits with whichthe intensity values are represented determines the dynamic range of theintensity values. In the example, it will be assumed that the intensityvalues are represented in eight bits, so that there are 255 possiblenon-zero values. Instead of underexposing the image frames, thelinear-scale ISO setting (also known as ASA number) of theimage-capturing device 5 can be increased by the same factor as theunderexposure factor. This results in increased noise levels in theindividual frames, which are reduced through the combination processespresented below.

In the embodiments described herein, a distinction is made between afirst set of arrays of intensity values representing associatedrespective image frames and a second set of arrays of intensity valuesrepresenting associated respective image frames. The distinction is madeon the basis of how the arrays are processed subsequent to capturing ofthe image frames.

In a first embodiment, depicted in FIG. 2, a first set 18 of arrays ofintensity values represents image frames at a relatively low spatialresolution, whereas a second set 19 of arrays of intensity valuesrepresents image frames at a relatively high resolution. Since thespatial resolution is proportional to the number of intensity values inthe arrays, it follows that the arrays in the first set contain fewervalues than those in the second set 19. This reduces the processingrequirements, which is advantageous, as will become clear.

It is noted that the amount of processing is already reduced merely bythe division of a sequence of arrays into the first set 18 and secondset 19, so that the fact that the first set represents image frames at alower resolution than the second set is an advantageous, but optionalfeature. Furthermore, it is not required, but efficient in terms ofprocessing, that the arrays that share a set all have the same number ofelements, i.e. that the image frames they represent each have the sameresolution. In the illustrated embodiment, a final array 20 of intensityvalues representing a combined image is formed on the basis of thearrays in the first and second set 18,19 only. In other embodiments,there may be a third set of arrays representing image frames in thesequence of successively captured image frames on which the combinedimage is based.

An object of the method illustrated in FIG. 2 is to scale the intensityvalues in the arrays of the first set 18 such that the final array 20contains intensity values that occupy the full dynamic range. The methodserves to prevent a situation in which all the intensity values in thefinal array are clipped at the highest of the 255 values afforded by aneight-bit representation.

In a first step 21, one or more arrays of intensity values in the firstset 18 of arrays are at least partially analysed. In one embodiment, theanalysis comprises the forming of a histogram of some or all of theintensity values. To reduce the processing effort required to generate ahistogram, only one value in every block of sixty-four values could beused.

If a significant number of intensity values lies above a thresholdvalue, then a mapping function is required, which mapping function isdetermined in a second step 22. The second step 22 is followed by a step23 in which a look-up table 24 is generated on the basis of the mappingfunction. For each of 255 intensity values, a scaled value is enteredinto the look-up table 24. Using a look-up table allows the mapping tobe carried out by the DSP 7, which is relatively efficient. Thus, theuse of a look-up table makes the methods presented herein quite suitablefor implementation in an imaging apparatus, such as the digital camera1.

Only the arrays of intensity values in the first set 18 are mapped (step25) to arrays of scaled intensity values in a set 26. Each intensityvalue is used as an index into the look-up table 24 to determine itsscaled value. It will be appreciated that, by scaling only the intensityvalues in the arrays forming the first set 18, a smaller look-up tableis required. Moreover, the number of look-up operations is much reduced.As will be seen, the final array 20 can still represent a combined imageat a higher resolution, because each intensity value in the final arrayis based on at least one intensity value in an array comprised in thesecond set 19. It is noted that the mapping function is applied directlyto the arrays of intensity values in the first set 18 in otherembodiments, so that the look-up table 24 is dispensed with.

The mapping function used to populate the look-up table 24 maps theintensity values from a first scale within a first range to a secondscale occupying a second, smaller range. In one embodiment, the upperlimit of the second scale is determined on the basis of at least twofactors. A first factor is the extent to which the intensity values ofthe arrays analysed in the first step 21 exceed a certain thresholdvalue. The second factor is based on the number of arrays of intensityvalues in the second set 19. More specifically, the threshold value isthe maximum value of the dynamic range for encoding the values in thefinal array 20, divided by the number of arrays in the first and secondsets 18,19. The mapping function is chosen to ensure that a substantialproportion of the intensity values in each of the arrays of the set 26of arrays of scaled intensity values remain below the threshold. Thesecond factor in this example is based on the ratio of the number ofarrays in the second set 19 to the number of arrays in the first set 18.The upper value of the second scale is obtained by reducing thethreshold by an amount corresponding to this ratio. Thus, the fact that,only the arrays in the first set 18 of the first and second sets 18,19are scaled is taken into account.

In an embodiment that is more efficient in its implementation, a fixedcurve or look-up table is used to determine the scaling in dependenceonly on the number of arrays of intensity values in the first and secondsets 18,19. Alternatively, a mapping function could be selected independence on the degree of overexposure or, equivalently, the factor bywhich the sensitivity of the image-capturing device 5 used to capturethe arrays of intensity values on which the arrays in the first andsecond sets 18,19 are respectively based has been increased.

In an advantageous embodiment, the first step 21 is preceded by a step(not shown), in which the first and second sets 18, 19 of arrays ofintensity values are obtained by obtaining a plurality of arrays ofintensity values for representing colour image frames in a first colourspace, and applying a transformation to a plurality of arrays of valuesin a second colour space, wherein, in the first colour space, an imageframe is represented by parameter value combinations, each parameterindicating the intensity of one of a plurality of colour components,whereas, in the second colour space, an image frame is represented byparameter value combinations, one parameter of the combinationindicating a hue and at least one of the other parameters beingindicative of light intensity. In the specific embodiment, arrays ofintensity values representing image frames in the RGB (Red Green Blue)colour space are transformed to respective arrays of parameter valuesrepresenting image frames in the HLS (Hue, Lightness, Saturation) colourspace. The RGB colour space is an additive colour space, wherein theintensity of each of the three colour components is encoded separately.If the entire method depicted in FIG. 2 is carried out in the RGB colourspace, then the method would in essence have to be carried out intriplicate. The first and second steps 21,22 would involve analysis ofthe three arrays that belong together in the sense that they represent acolour component of the same image frame. At least the scaling step 25involves scaling three arrays of intensity values per image frame. Inthe HLS colour space, an image is represented by the parametercombination Hue, indicating the relative strengths of three colourcomponents, Saturation, providing a scale from a grey level to a fullcolour, and Lightness (also called Luminance) correspondingsubstantially to the average intensity of the colour components. Onlythe arrays of Lightness values in the first set 18 are scaled. It isnoted that the HSV (Hue, Saturation, Value) colour space is usable as analternative to the HSL colour space, and that the CMYK and YUV colourspaces are alternatives to the RGB colour space.

As mentioned, each intensity value in the final array 20 is based on atleast one intensity value in an array comprised in the second set 19 ofarrays of intensity values. In the embodiment illustrated in FIG. 2,this is assured by summing corresponding pixel values of each of thearrays in the set 26 of arrays of scaled intensity values

In order to obtain a high-resolution combined image, a set 27 ofresolution-adjusted arrays is generated (step 28). In this step 28, thespatial resolution of the arrays in the set 26 of arrays of scaledintensity values is adjusted by a multiplication factor, and isincreased. An alternative would be to decrease the resolution of theimage frames represented by the arrays in the second set 19. One way ofincreasing the spatial resolution of the image frames represented by thearrays in the set 26 of arrays of scaled intensity values is tointerpolate between the intensity values in the arrays of scaledintensity values.

The final array 20 is obtained by summing (step 29) an intensity valuefrom each of the arrays in the set 27 of resolution-adjusted arrays andvalue from each of the second set 19 of arrays. Intensity valuescorresponding to the same pixel in the scene represented by the imageframes are added.

To take account of camera shake, an additional step (not shown) iscarried out to correct the image frames. The correction may be carriedout prior to the first step 21 shown in FIG. 2, so that the arrays ofthe first and second set 18,19 are the result of the correctionoperation. In this case, each array in the first and second sets 18,19is based on an array of intensity values obtained by the image-capturingdevice 5 and corrected in accordance with a motion vector. The motionvector describes the motion of the camera 1 between the points in timeat which the arrays of intensity values were obtained by theimage-capturing device. It is based on data obtained from the motionsensor 15 or based on an analysis of the captured image frames using amethod described more fully in international patent applicationPCT/EP04/051080, which is hereby incorporated by reference. In thatapplication, a method is described that includes calculating a motionvector representing at least a component indicative of relative movementof at least a part of successive image frames in a sequence of imageframes, wherein the step of calculating the motion vector includes astep of determining at least a first term in a series expansionrepresenting at least one element of the motion vector, which stepincludes an estimation process wherein at least the part in each of aplurality of the image frames is repositioned in accordance with thecalculated motion vector. The estimation process includes calculation ofa measure of energy contained in an upper range of the spatial frequencyspectrum of the combined image and the step of determining at least thefirst term includes at least one further iteration of the estimationprocess to maximise the energy.

In an alternative embodiment, the image frames are aligned using amethod known per se by the name of Random Sample Consensus (RANSAC).This method is suitable where there is sufficient light to capture imageframes.

FIG. 3 illustrates a variant of the method shown in FIG. 2. Thisembodiment is also based on a first set 30 of arrays of intensity valuesand a second set 31 of arrays of intensity values. Each intensity valueis a pixel value, corresponding to the light intensity of an associatedcolour component over an area associated with a pixel. What has beenstated above regarding the first and second set 18,19 shown in FIG. 2applies equally to the first and second set 30,31 shown in FIG. 3.Again, this description will assume that the arrays of intensity valuein the first set 30 of arrays represent image frames at a lowerresolution than the arrays in the second set 31.

A first step 32 in the method of FIG. 3 corresponds to the first step 21shown in FIG. 3. In a subsequent step 33, a mapping function is againdetermined in order to map the intensity values of the arrays in thefirst set 30 from a scale occupying a first range to a second scaleoccupying a second range. The mapping function is determined on thebasis of at least parts of some or all of the arrays in the first set30. It is determined in substantially the same way as in the embodimentof FIG. 2. Similarly, a look-up table 34 is created in a step 34following the step 33 of determining the mapping function. The look-uptable 34 is used (step 36) to generate a set 37 of arrays of scaledintensity values, in which each array is based on a corresponding arrayin the first set 30 of arrays of intensity values.

The variant of FIG. 3 differs from the one shown in FIG. 2, in that atransformation to the spatial frequency domain is carried out in anotherstep 38 subsequent to the scaling step 36. This transformation step 38is implemented using a Discrete Cosine Transform (DCT) in theillustrated example. The set 37 of arrays of scaled intensity values isthe basis for a first set 39 of arrays of DCT coefficients. The secondset 31 of arrays of intensity values is the basis for a second set 40 ofarrays of DCT coefficients. It is observed that the DCT transform ispart of the JPEG (Joint Photographic Experts Group) compressionalgorithm, and that it is advantageous to implement such an algorithm ina special-purpose processor, such as the DSP 7 or co-processor 11. Atransformation from the RGB colour space to the HLS colour space is alsopart of the JPEG algorithm, so that this feature is also applied toadvantage in the embodiment illustrated in FIG. 3. The transformationbetween colour spaces has been detailed above.

A summation step 41 is carried out in the spatial frequency domain toobtain a final array 42 of DCT coefficients. The final array 42 forms anarray of intensity values representing a combined image, since eachcoefficient is indicative of the intensity level of a spatial frequencycomponent, and the set of spatial frequency components contains all theinformation necessary to render the combined image. The low-frequencycoefficients of the final array 42 are obtained by summing thelow-frequency coefficients of each array in the first set 39 of arraysof DCT coefficients and the low-frequency coefficients of each array inthe second set 40 of arrays of DCT coefficients. The high-frequencycoefficients are obtained by summing the high-frequency coefficients ofeach array in the second set 40 of arrays of DCT coefficients. Sincethese higher-order coefficients are absent in the (smaller) arrays ofthe first set 39 of arrays of DCT coefficients, only some of theintensity values in the final array 42 of DCT coefficients are obtainedon the basis of both the first and second set 30,31 of arrays ofintensity values. The summation step 41 is preferably implemented so asto take account of the differing number of addends used to obtain eachcoefficient in the final array 42.

An Inverse Discrete Cosine Transformation (IDCT) 43 results in an array44 of intensity values in the spatial domain. Both the transformationstep 38 and the IDCT 43 are advantageously carried out by theco-processor 11 in the digital camera 1.

FIG. 4 illustrates an embodiment for simplifying the determination ofthe mapping function from the first scale to the second scale, as wellas simplifying the scaling step. It operates on the basis of a first set45 of arrays of pixel values and a second set 46 of arrays of pixelvalues.

A first sum array 47 is formed in a first step 48. On the assumptionthat the arrays in the first set 45 represent respective image frames atthe same resolution, each intensity value in the first sum array isobtained by summing the corresponding intensity values from each of thearrays in the first set 45. If the resolutions are not the same,interpolation may be carried out first, or the arrays representinghigher-resolution image frames may be reduced to correspond to a commonresolution. The first sum array 47 is also suitable for representing animage frame, albeit one based on a plurality of preceding image frames,and forms a set of arrays consisting of one member. In alternativeembodiments, a plurality of sum arrays could be formed, each based on asubset of arrays in the first set 45, with the plurality of sum arraysforming a first set in the terminology used herein.

The first sum array 47 of intensity values is analysed (step 49) todetermine a mapping function for mapping a first scale occupying a firstrange to a second scale occupying a second range. As described before,the analysis advantageously comprises the forming of a histogram of someor all of the intensity values, i.e. DCT coefficients. Again, this maybe carried out using one value per block of intensity values within thefirst sum array. However, the embodiment of FIG. 4, because only thefirst sum array 47 is analysed, allows for a more involved analysis ascompared to embodiments in which a number of arrays of intensity valueshave to be analysed.

If a significant number of intensity values lies above a thresholdvalue, then a mapping function is required. A look-up table 50 isgenerated (step 51) on the basis of the mapping function. For each of,for example, 255 intensity values, a scaled value is entered into thelook-up table 50.

Only the first sum array 47 of intensity values is mapped (step 52) to ascaled first sum array 53. Preferably, the arrays in the first set 45 ofintensity values represent image frames at a lower resolution than thearrays in the second set 46 of arrays. Even if this is not the case, itis still feasible to generate a first sum array 47 representing acombined image frame at a lower resolution than that at which imageframes are represented by the arrays in the second set 46 of arrays ofintensity values. Thus, the number of look-up operations is keptrelatively small.

As before, the mapping function used to populate the look-up table 50maps the intensity values from a first scale within a first range to asecond scale occupying a second, smaller range. The upper limit of thesecond scale is again determined on the basis of at least two factors. Afirst factor is the extent to which the intensity values of the firstsum array 47 exceed a certain threshold value. The second factor isbased on the number of arrays of intensity values in the second set 46.More specifically, the threshold value is the maximum value of thedynamic range for encoding the values in the first sum array 47. Themapping function is chosen to ensure that a substantial proportion ofthe intensity values in the scaled first sum array 53 remain below thethreshold. The second factor in this example is based on the ratio ofthe number of arrays in the second set 46 to the number of arrays in thefirst set 45. The upper value of the second scale is obtained byreducing the threshold by an amount corresponding to this ratio. Thus,the fact that, only the first sum array 47 is scaled, and not also thearrays in the second set 46 of arrays of intensity values, is taken intoaccount.

Scaling only the first sum array 47 reduces even further the number oflook-up operations. Nevertheless, it would be possible to analyse thefirst sum array 47 to derive a mapping function for scaling theindividual arrays in the first set 45 of arrays, which are then addedafter having been scaled. Alternatively, it would be possible to analysethe individual frames in the first set 45 of arrays of intensity values,in order to derive a mapping function for scaling the first sum array47. The effect of scaling the first sum array 47 is to reduce the amountof noise that propagates to a final array 54 of intensity valuesrepresenting a combined image.

The final array 54 of intensity values represents a combined image at ahigher resolution than the scaled first sum array 53. For this reason,the latter is processed (step 55) to obtain a resolution-adjusted scaledfirst sum array 56. Again, interpolation is a method by which theintensity values in the resolution-adjusted scaled first sum array 56can be obtained.

The final array 54 is obtained in a final step 57. In this step 57, eachintensity value in the final array 54 of intensity values is obtained bysumming an intensity value from the resolution-adjusted scaled first sumarray 56 and the corresponding respective intensity values from each ofthe arrays in the second set 46 of arrays of intensity values. It willbe apparent that the final array 54 is thus formed of intensity valuesthat are each based on at least one intensity value in an array in thesecond set 46 of arrays of intensity values, to achieve ahigh-resolution representation of the combined image.

FIG. 5 shows a variant in which calculation is largely carried out inthe spatial frequency domain, and which does not necessarily requireinterpolation or another process for increasing the resolution at whichan image frame is represented. The variant illustrated in FIG. 5commences with a DCT operation 58. The DCT operation 58 is used toobtain a first set 59 of arrays of DCT coefficients for representing aset of corresponding image frames at a first resolution. This first set59 is based on a set 60 of arrays of pixel values encoding the imageframes in the spatial domain as opposed to the spatial frequency domain.A second set 61 of arrays of DCT coefficients is based on a second set62 of arrays of pixel values encoding image frames in the spatial domainat a second resolution. In this example, it will again be assumed thatthe second resolution is higher than the first resolution.

In a subsequent step 63, the arrays in the first set 59 of arrays of DCTcoefficients are processed to obtain a first sum array 64. Each DCTcoefficient in the first sum array 64 is obtained by summing thecorresponding DCT coefficients in the respective arrays of the first set59.

The first sum array 64 is analysed to determine (step 65) a mappingfunction mapping the DCT coefficients in the first sum array 64 from afirst scale occupying a first range to a second scale occupying asecond, preferably smaller, range. This step 65 is carried out using anyof the methods outlined above with regard to the corresponding steps 22,33, 49 in the methods of FIGS. 2-4. Subsequently (step 66) a look-uptable 67 is created on the basis of the mapping function.

The mapping functions is based at least partly on the number of arraysin the second set 61 of arrays of DCT coefficients. This is done becauseonly the DCT coefficients in the first sum array 64 are mapped from thefirst scale to the second scale (step 68), whereas those in the arraysforming the second set 61 of arrays of DCT coefficients are not. Theresult of the scaling carried out in this step 68 is a scaled first sumarray 69.

The scaled first sum array 69 and the arrays in the second set 61 ofarrays of DCT coefficients are summed in a step 70 similar to thesummation step 41 in the embodiment illustrated in FIG. 3. A final array71 of DCT coefficients is obtained. The lower-order DCT coefficients inthe final array 71 of DCT coefficients are each obtained by summing thelower-order coefficients of the scaled first sum array 69, which isbased on the first sum array 64, and the corresponding lower-ordercoefficients of the arrays of the second set 61 of arrays of DCTcoefficients. The higher-order DCT coefficients in the final array 71are obtained by summing the corresponding higher-order coefficients inthe arrays comprised in the second set 61 of arrays of DCT coefficientsonly. Thus, the final array 71 of DCT coefficients is suitable forrepresenting the combined image at a relatively high resolution, atleast higher than that of the image frames represented by the first sumarray 64.

An inverse DCT operation 72 transforms the final array 71 of DCTcoefficients into a final array 73 of pixel values, each correspondingto a light intensity over an area occupied by a pixel in the combinedimage.

The invention is not limited to the described embodiments, which may bevaried within the scope of the accompanying claims. In particular, themethods outlined herein are suitable for partial or complete executionby another type of image processing system than the digital camera 1.For example, a general-purpose personal computer or work station maycarry out the method on the basis of a first set of arrays of pixelvalues and a second set of arrays of pixel values in a sequence ofarrays captured in rapid succession by the digital camera 1 and storedin the storage device 8. Processing of the arrays for relative alignmentof at least the region of overlap between the image frames representedby them is an advantageous feature of each embodiment.

1. Method of forming a combined image based on a plurality of imageframes, including: obtaining a first set (18;30;47;64) of at least onearray of intensity values for representing an image frame at aresolution corresponding to the number of intensity values in the array,and obtaining a second set (19;31;46;61) of at least one array ofintensity values for representing an image frame at a resolutioncorresponding to the number of intensity values in the array, whereinthe combined image is represented by a final array (20;42;54;71) ofintensity values, wherein at least some of the intensity values in thefinal array (20;42;54;71) are each obtained by executing a step(29;41;57;70) of summing an intensity value from each of at least onearray (27;39;56;69) of intensity values based on at least one array ofintensity values in only the first set (18;30;47;64) and an intensityvalue from each of at least one array (19;40;46;61) of intensity valuesbased on at least one array of intensity values in only the second set(19;31;46;61), wherein, prior to executing the summing step(29;41;57;70), only the intensity values of the arrays in the first set(18;30;47;64) are mapped from a scale within a first range to a scalewithin a second range.
 2. Method according to claim 1, includingobtaining a first set (18;30;47;64) and a second set (19;31;46;61)arranged such that the image frames represented by the arrays in thefirst set (18;30;47;64) are represented at lower resolutions than theimage frames represented by the arrays in the second set (19;31;46;61).3. Method according to claim 2, wherein the at least one array (39;69)of intensity values based on at least one array of intensity values inonly the first set (30;64) contains coefficients in the spatialfrequency domain, wherein the at least one array (40;61) of intensityvalues based on at least one array of intensity values in only thesecond set (31; 61) contains coefficients in the spatial frequencydomain and wherein the intensity values in the final array (42;71) areformed by coefficients in the spatial frequency domain, wherein at leastone lower order coefficient in the final array (42;71) is obtained bysumming an intensity value from each of the at least one array (39;69)of intensity values based on at least one array of intensity values inonly the first set (30;64) and an intensity value from each of at leastone array (40;61) of intensity values based on at least one array ofintensity values in only the second set (31;61), wherein at least onehigher order coefficient in the final array (42;71) is obtained on thebasis of only arrays (40;61) of intensity values based on the second set(31;61).
 4. Method according to claim 2, wherein at least some of thearrays of intensity values in the first and second set are obtained byreading out measurement values from an image-capturing device comprisingan array of light-sensitive cells, wherein each intensity value in thefinal array (20;42;54;71) is based on at least one intensity value in anarray comprised in the second set.
 5. Method according to claim 1,including determining an upper limit of the second range at least partlyin dependence on the number of arrays of intensity values in the secondset (19;31;46;61).
 6. Method according to claim 1 wherein at least oneof the arrays of intensity values in the first set (47;64) is obtainedby obtaining a plurality of arrays (45;60) of intensity values forrepresenting an image frame at a resolution corresponding to the numberof intensity values in the array, and by summing an intensity value fromeach of the plurality of arrays (45;60) to obtain a correspondingintensity value in the at least one array in the first set (47;64). 7.Method according to claim 1, wherein at least one of the arrays ofintensity values in the first set (18;30;47;64) is obtained by obtaininga plurality of arrays (45;60) of intensity values for representing animage frame at a resolution corresponding to the number of intensityvalues in the array, wherein the method further includes summing anintensity value from each of the obtained plurality of arrays (45;60) toobtain a corresponding intensity value in an intermediate combined array(47;64), and determining an upper limit of the second range at leastpartly in dependence on at least one intensity value in the intermediatecombined array (47;64).
 8. Method according to claim 1, wherein at leastthe arrays of intensity values in the first set are obtained byobtaining a plurality of arrays of intensity values for representingcolour image frames in a first colour space, and applying atransformation to a plurality of arrays of values in a second colourspace, wherein, in the first colour space, an image frame is representedby parameter value combinations, each parameter indicating the intensityof one of a plurality of colour components, whereas, in the secondcolour space, an image frame is represented by parameter valuecombinations, one parameter of the combination indicating a hue and atleast one of the other parameters being indicative of light intensity.9. System for processing arrays of intensity values, each array beingsuitable for representing an image frame at a resolution correspondingto the number of intensity values in the array, wherein the system isconfigured to retrieve a first set (18;30;47;64) of at least one arrayof intensity values and a second set (19;31;46;61) of at least one arrayof intensity values, the arrays in the first set (18;30;47;64) andarrays in the second set representing respective image frames, and toform a final array (20;42;54;71) of intensity values representing acombined image, wherein the system is configured to obtain each of atleast some of the intensity values in the final array (20;42;54;71) byexecuting a step of summing an intensity value from each of at least onearray (27;39;56;69) of intensity values based on at least one array ofintensity values in only the first set (18;30;47;64) and an intensityvalue from each of at least one array of intensity values based on atleast one array (19;40;46;61) of intensity values in only the second set(19;31;46;61), and wherein the system is configured to map, prior toexecuting the summing step, only the intensity values of the arrays inthe first set (18;30;47;64) from a scale within a first range to a scalewithin a second range.
 10. (canceled)
 11. Imaging apparatus, e.g. adigital camera (1), comprising a processor (7,9,11) and at least onestorage device (8) for storing a plurality of arrays of intensityvalues, wherein the imaging apparatus is configured to execute a methodaccording to claim
 1. 12. Computer program, including a set ofinstructions capable, when incorporated in a machine-readable medium, ofcausing a system (1) having information processing capabilities toperform a method according to claim 1.